Add local HA operator status - #887

Reviewable diff: +795/-78 across 16 files (excludes generated, test, and story files).

Summary

Gives operators a trustworthy answer to "is this HA cluster healthy, and would failover succeed right now?" without exposing cluster internals. Each Fleet host serves a redacted, local-only HA status document, and a new fleet-ha status command combines it with independent probes of etcd, Patroni, PostgreSQL, the VIP, and both Fleet hosts to produce a machine-readable readiness verdict. The public /health response now carries the running release version in an X-Proto-Fleet-Version header, which the readiness check uses to detect mixed-version peers.

Stack: This is PR 1 of 6, targeting main: #887 -> #888 (install on clean Linux hosts) -> #889 (clean-install qualification) -> #890 (passive host updates) -> #891 (updates through bounded failover) -> #892 (adjacent-release update qualification). This PR only establishes operator visibility; installation, qualification, and updates land in the descendants, which use fleet-ha status as their final readiness gate.

How it works

Inside fleetd. The HA coordinator now tracks structured observation state instead of a free-text last error: whether an observation is available, and a FreshUntil horizon computed from the lease interval but never extending past the DCS proof deadline of the last writer observation. Losing the lease acquisition race to the peer is reclassified as a healthy outcome: a new SQL query (ClassifyFleetRuntimeLeaseAcquisition) distinguishes genuine contention from cluster identity or writer mismatches, and the coordinator records a contended acquisition as a valid passive observation after completing its closing DCS proof. Runtime.Status composes this into the redacted operator contract: a coarse role (active, passive, initializing, degraded), observation freshness, endpoint health, and machine-readable reason codes. A passive node that still owns the VIP reports degraded, not passive.

Exposure boundaries. The status document is served at /health/ha only when HA is enabled, and fleetd refuses to start in HA mode unless it listens on exactly 127.0.0.1:4000, so the document cannot be exposed remotely by misconfiguration. The VIP nginx additionally returns 404 for /api-proxy/health/ha. Two signals are deliberately public through the VIP proxy: /health gains the X-Proto-Fleet-Version header, and a new /health/passive endpoint reports only whether the process is ready to take over (fresh observation, passive state, no VIP ownership).

The CLI. fleet-ha status [node.env] runs on ha-a or ha-b. It reads the local /health/ha document, then runs five parallel control-path probes, each bounded to ~2 seconds, using the host's on-disk secrets and the read-only fleet-observer etcd credential:

1. etcd membership: probes each of the three endpoints individually and cross-checks cluster/member identities, detecting split or cloned members that a single pooled client would hide.
2. Patroni roles: requires exactly one primary and one fully caught-up synchronous replica across the two database hosts.
3. Writer observation: replays the same Observer.Observe validation fleetd uses to acquire ownership (DCS leader, writable PostgreSQL identity, Patroni role and timeline agreement, live lease), over a fresh single pinned DB connection built from the on-disk DB_DSN (revalidated with ValidateHA).
4. VIP: the virtual IP must answer as the active Fleet.
5. Per-host Fleet roles: dials each database host directly while keeping the VIP TLS identity, requiring exactly one active and one live passive, both reporting the same release version.

The report derives control_ready (the cluster works now) and failover_ready (losing the active node would be survivable), each explained by reason codes (etcd_quorum_unavailable, writer_unavailable, fleet_version_mismatch, ...). Output is JSON; the command exits nonzero unless failover_ready, making it directly usable as a scripted gate.

sequenceDiagram
    participant OP as Operator
    participant CLI as fleet-ha status
    participant FD as fleetd (127.0.0.1:4000)
    participant DEP as etcd / Patroni / PostgreSQL / VIP / peer Fleet
    OP->>CLI: run on ha-a or ha-b
    CLI->>FD: GET /health/ha
    FD-->>CLI: redacted runtime status
    CLI->>DEP: 5 parallel read-only probes (~2s each)
    DEP-->>CLI: per-dependency results
    CLI-->>OP: JSON report, exit 0 only if failover_ready

flowchart TD
  RT["Coordinator snapshot: observed, FreshUntil (clamped by DCS proof deadline), state"] --> ST["Runtime.Status: role + observation + endpoint + reason codes"]
  ST --> EP["/health/ha (local only)"]
  EP --> CLI["fleet-ha status"]
  PR["etcd + Patroni + writer replay + VIP + peer Fleet probes"] --> CLI
  CLI --> CR["control_ready: quorum, one primary, writer probe, VIP, one active Fleet"]
  CR --> FR["failover_ready: control_ready plus full etcd redundancy, one sync replica, live passive, matching versions, local runtime current"]

Areas of the code involved

Key technical decisions & trade-offs

• Redacted status contract: coarse role/observation/endpoint enums plus reason codes, instead of exposing coordinator internals. LastError was removed from Snapshot entirely so raw error strings (DSNs, hostnames) cannot reach any reportable surface.
• Locality enforced in code, not just deployment: fleetd hard-fails HA startup unless bound to 127.0.0.1:4000, and the VIP nginx 404s the path as a second layer, rather than relying on either alone.
• Lease contention classified in SQL: one query returns the acquisition outcome, chosen over a read-then-decide sequence in Go that would race with the peer.
• Freshness bounded by the DCS proof deadline: a node can never claim its cluster view is fresh beyond the window its linearizable etcd proof is valid for, rather than using a wall-clock interval alone.
• Independent verification over self-reports: the writer probe replays the runtime's own observation logic with on-disk credentials, catching stale DCS leader records, timeline divergence, and broken rotated secrets that component health endpoints cannot see.
• status is a readiness gate, not a passive viewer: it always probes, always prints JSON, and exits nonzero unless failover-ready. There is no flag surface to keep qualified.
• Version-mismatch detection via the public header: failover_ready requires both Fleet hosts to report the same X-Proto-Fleet-Version, so a mid-upgrade cluster is never declared safe to fail over.
• Hardened probe transport: every probe client disables proxies, rejects redirects, clones TLS config, and uses ~2s timeouts; all cluster access uses the read-only fleet-observer role.

Testing & validation

• Unit tests cover the status derivation table (roles, freshness, endpoint states), coordinator contention and proof-deadline clamping, the health handlers, CLI argument handling and exit behavior, and the pure probe summarizers (etcd identity dedup/split detection, fleet redundancy, version matching, probe DSN validation and rejection).
• A store integration test exercises ClassifyFleetRuntimeLeaseAcquisition against a real database.
• deployment-files/ha/tests/test-profile.sh asserts the nginx 404 for /api-proxy/health/ha.
• Not covered here: live three-host behavior (real etcd, Patroni, keepalived, and actual failover) is deferred to the qualification PR (Define clean-install HA qualification #889). The network probes themselves are validated through their pure summarizers, not against live dependencies.